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ABSTRACT 

In  field  tests  to  compare  the  observability  of  combat  vehicles,  the  test  designer  must  select  the  optimum  number  of 
observation  opportunities  in  order  to  bailee  collecting  enou^  data  to  draw  valid  conclusions  against  th§  high  cost  of 
supporting  vehicles  and  personnel  at  a  test  site.  The  test  designs,  however,  generally  lacks  key  parameters  for  the 
efBcient  design  of  the  test.  Namely,  the  designer  lacks  die  detectim  probabilities  of  the  vehicles  at  each  range.  The 
standard  deviation  of  the  difference  in  detection  probability  depends  upon  the  detection  probability  itself.  Therefore,  the 
test  designer  must  select  the  number  of  observations  for  each  range  ba^  upon  the  consavative  assumptim  that  the 
probabilities  are  near  50%,  the  probability  for  the  maximum  standard  deviation.  In  a  previous  paper,  I  presented  an 
iterative  technique  of  test  design  in  order  to  improve  the  efficiency  of  observability  tests.  In  this  paper,  I  present  the 
results  of  a  Monte  Carlo  evalulation  of  this  iterative  technique. 


1.  THE  EXPERIMENTAL  SITUATION 

Figure  1  illustrates  a  typical  test  setup  for  a  test  of  observability.  Observers  are  statimed  at  a  fixed  site  and  attempt  to 
detect  a  vehicle  in  their  field  of  view.  For  each  observation  opportunity,  the  test  personnel  record  the  number  of 
detections.  In  analyzing  die  data,  the  analyst  groups  the  observations  into  range  bins  and  compares  the  proportion  of 
detections  for  each  test  vehicle. 

For  such  a  field  test,  the  test  designer  must  select  the  optimum  number  of  observation  opportunities  at  each  range.  The 
designer  must  balance  collecting  enough  data  to  draw  valid  conclusions  against  the  high  cost  of  supporting  vehicles  and 
persoimel  at  a  test  site. 


2.  DESIGNING  THE  EXPERIMENT 

The  test  designer  selects  the  number  of  observations  in  orda  to  adiieve  two  objectives.  First,  the  number  of  observations 
must  be  large  aiough  so  that  the  probabilities  of  errors  are  below  the  desired  level.  And,  second,  the  number  of 
observations  must  be  as  small  as  possible  to  minimize  testing  time  and  cost. 

Table  I  illustrates  the  types  of  errors  that  can  occur  in  drawing  conclusions  from  an  observation  test  The  goal  of  an 
observation  test  is  to  ju^e  whether  the  observed  difference  in  detectim  probability,  Pd ,  is  unusual  enough  to  reject  the 
null  hypothesis  that  Vdiicle  A  and  Vehicle  B  have  the  same  Pd .  If  the  analyst  concludes  that  the  null  hypothesis  caimot 
be  rejected,  then  either  the  analyst  has  made  a  correct  decision  or  a  Type  II  citot.  On  the  other  hand,  a  decision  to  reject 
the  null  hypothesis  will  be  eitho-  correct  or  a  Type  1  error.  In  teams  of  countermeasure  effectiveness,  a  Type  I  error  is  an 
erroneous  conclusion  than  an  ineffective  countermeasure  is  effective,  while  a  Type  II  error  miss  judges  an  effective 
countermeasure. 

In  a  previous  papCT  [1],  I  discussed  the  number  of  observation  opportunities  required  to  control  die  probability  of  these 
errors.  For  the  case  of  an  average  Pd  of  0.5,  Figure  2  plots  the  number  of  observations  required  to  maintain  the 
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probability  of  Type  I  and  Type  II  erro's  at  less  tiian  5%.  For  example,  85  observations  of  each  vehicle  are  adequate  for 
an  observed  0.15  difference  to  be  significant  with  less  than  5%  chmce  of  a  Type  I  error.  On  the  otiier  hand,  for  less  than 
5%  chance  of  committing  a  Type  II  error  when  the  underlying  probabilities  differ  by  0. 1 5,  the  requiremait  is  3 16 
observation  opportunities  per  vehicle. 


3,  AN  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 

In  a  previous  paper  [2],  I  presented  an  iterative  technique  to  reduce  the  number  of  observation  opportunities  required  for 
a  test.  Observation  data  is  described  by  the  binary  distribution.  But  the  binary  distribution  has  the  property  that  the 
standard  deviation  depends  upcxi  the  probability.  If  p  is  the  probability  and  N  is  the  number  of  trials,  then  the  standard 
deviation  of  the  number  of  successes,  o,  is  given  by 

a  =  ^Np(l-p)  (1) 

And  the  standard  deviation  of  the  propcation  of  successes  is  given  by 


£1= 

N  \  N 


(2) 


Figure  3  plots  Equation  2  for  an  N  of  100, 200  and  300.  Figure  4  plots  file  same  equation  normalized  to  its  maximum  at 
p  =  0.5. 

Figure  5  fiirtha  shows  how  the  number  of  trials  change  with  probability  if  the  standard  deviation  is  held  fixed.  For 
example,  in  comparison  to  a  probability  of  0.5,  caily  60%  as  many  trials  are  need  at  a  probability  of  0. 1. 

To  use  this  property  of  the  binary  distributiai,  the  test  designer  modifies  the  test  matrbc  as  the  test  progresses.  Early  test 
results  yield  estimates  of  the  probability  of  detection  for  each  vdiicle  at  each  range.  Based  on  these  estimates,  the  test 
designer  reallocates  the  number  of  observations  among  file  ranges,  improving  fiie  efficiency  of  the  test  design.  Figures  5 
and  6  illustrates  a  sample  of  the  improvement  in  efficiency  that  this  technique  can  achieve.  Initially  the  test  designer 
would  have  selected  the  number  of  observations  at  each  test  range  by  assuming  the  worst-case  value  of  Pa=  0.5.  But  as 
file  test  progresses,  the  designer  would  use  estimates  of  Pj  fi'can  the  early  measures  to  redesign  the  test.  For  example,  at 
range  20,  file  designer  would  reduce  the  number  of  observations  fi’om  314  to  251.  Overall  in  this  sample  test,  the  test 
designer  would  reduce  the  number  of  observations  from  1570  to  1 193,  a  reduction  of  24%. 


4.  MONTE  CARLO  EVALUATION 

Simulation  of  the  example  of  Figures  5  illustrates  the  speed  of  convergence  of  this  iterative  technique.  Figure  7  shows 
how  the  number  of  observations  pCT  day  at  each  range  would  change  as  the  itwative  technique  is  applied.  Initially,  the 
assumed  50  observation  per  day  would  be  divided  evenly  among  the  5  ranges.  As  the  test  progresses,  more  observations 
would  be  allotted  to  the  range  with  a  probability  of  0.5  and  fewer  to  the  otiiCT  ranges.  Figure  8  plots  the  difference 
between  the  adjusted  number  of  runs  per  day  for  each  range  and  the  number  afta'  100  observations.  After  30 
observations  at  each  range,  the  adjustments  are  mincx. 

5.  CONCLUSION 

In  this  papa-,  I  have  presented  an  iterative  technique  of  using  early  test  results  in  an  observation  test  to  improve  the 
overall  efficiency  of  the  test  design.  Moreover,  a  simulaticxi  showed  that  the  technique  yields  quickly  converging 
adjusted  values  for  the  number  of  observatirais  at  each  range  based  on  estimates  of  probabilities  of  detection 
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Possible  Outcomes  of  Hypothesis  Testing 


Is  Vehicle  A  Less  Detect^le 
Than  Vehicle  B? 


Decision; 

No 

Yes 

Accept  Null  Hypothesis 

Correct 

T ype  II  Error 

Accept  Alternative  Hypothesis 

Type  1  Error 

Correct 

Table  1.  Definitions  of  Type  I  and  Type  II  errors. 


Test  with  Fixed  Observers 


Fixed  Observers 


Figure  1.  Expo-imental  setup  for  an  observer  test. 
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Figure  2.  Number  of  observation  opportunities  required  to  meet  test  criteria. 
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Figure  3.  Standard  deviatim  of  a  binary  distribution  for  100, 200  and  300  trials. 
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Figure  4.  Standard  deviaticHi  of  a  binary  distribution  normalize  to  probability  of  0.5. 
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Figure  7.  Adjustment  of  number  of  observations  at  eadi  range  as  sample  test  progresses. 
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Figure  8.  CcHivergence  of  the  number  of  observations  pa*  day  for  each  range. 


